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1.  INTRODUCTION 

Health  care  team  performance  is  critical  to  the  provision  of  safe,  efficient,  and  effective  care.  Team  adaptability 
is  necessary  for  effective  team  performance  and  is  especially  critical  for  trauma  teams,  whose  members  must 
anticipate  change  and  rapidly  coordinate  effective  responses.  Teams  that  are  not  highly  adaptive  function  in  a 
reactive  mode  that  is  fraught  with  potential  safety  and  error  risks.  Rigorously  designed  computer-based 
simulation  systems  have  the  potential  to  support  active  learning  experiences  and  improve  adaptability  and 
performance  in  individuals  and  teams.  This  technology  has  the  potential  to  link  individuals,  teams,  and  units 
together  for  the  purpose  of  engaging  in  common  training  exercises.  However,  without  the  proper  supporting 
design  elements,  these  simulations  are  ineffective  and  inefficient  training  tools.  Current  health  care  team 
training  models  and  strategies  do  not  specifically  leverage  the  training  design  elements  and  assessment-driven 
feedback  mechanisms  that  improve  team  performance  in  highly  dynamic  settings.  The  goal  of  the  proposed 
project  is  to  improve  health  care  team  adaptability  and  patient  safety  by  providing  the  necessary 
conceptual  framework  and  assessment  mechanism  to  support  the  design  and  implementation  of  highly 
effective  simulation-based  team  training  with  embedded,  adaptive  guidance.  This  project  is  organized  into  the 
following  Aims: 

Aim  la.  Develop  a  team  training  design  architecture  to  support  simulation-based  training 

/assessment  systems  capable  of  developing  adaptive  expertise  in  healthcare  teams 

Aim  1b.  Develop  evidence-based  guidelines  and  recommendations  for  the  development  of 
embedded,  adaptive  feedback  and  performance  assessments 

Aim  2.  Develop  and  refine  a  predictive  model  of  trauma  team  performance  and  outcomes  for  use  in 
an  adaptive  guidance/feedback  system 

The  outcomes  from  this  research  will  provide  the  fundamental  knowledge,  both  conceptual  and  operational,  to 
support  the  development  of  simulation-based  team  training  systems  with  embedded  guidance.  Our  long-term 
goal  is  to  optimize  health  care  team  performance  and  adaptability  through  rigorous  training  design. 


2.  KEYWORDS  AND  ABBREVIATIONS 

Healthcare  team 

Trauma 

Trauma  teams 

Team  training 

Teamwork 

Adaptability 

Adaptive  performance 

Leadership 

Simulation 

Modeling 

Bayesian  belief  networks  (BBN) 
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3.  ACCOMPLISHMENTS 


3a.  What  were  the  major  goals  of  the  project  (organized  by  Aim)? 

Aim  la.  Develop  a  team  training  design  architecture  to  support  simulation-based  training/assessment 
systems  capable  of  developing  adaptive  expertise  in  health  care  teams 
Aim  1b.  Develop  evidence-based  guidelines  and  recommendations  for  the  development  of  embedded, 
adaptive  feedback  and  performance  assessments 

The  primary  outcome  of  Aim  la  is  a  conceptually  and  methodologically  sound  training  design  architecture  that 
supports  the  development  and  integration  of  team  training  and  automated  assessment  technologies  in 
simulation  environments.  The  primary  outcome  of  Aim  1b  is  a  set  of  best  practice  guidelines  and 
recommendations  for  the  design  and  incorporation  of  adaptive,  embedded  feedback  (guidance)  into  simulation- 
based  team  training.  The  tasks,  timeline,  and  status  of  each  step  associated  with  Aims  la  and  1b  are 
summarized  in  the  table  below. 


Aims  la  and  1b:  Major  Goals  and  Tasks 


Aims  la  and  1b  Tasks 

Timeline 

(Months) 

Status 

Task  1 :  Project  Start-up 

Establish  subcontracts  to  enable  purchasing. 

0-3 

COMPLETED 

Local/Site  IRB  application  submissions 

0-3 

All  IRB  submissions  have  been  completed  and  the  project  has 
been  awarded  exempt  status  by  each  institution.  COMPLETED 

Assembly  of  subject  matter  expert  panel 

0-3 

Subject  matter  experts  have  been  invited  and  the  panel  now 
contains  experts  from  emergency  medicine,  simulation,  trauma 
surgery,  and  nursing.  Individuals  were  chosen  for  their  expertise 
and  to  ensure  geographical  representation.  COMPLETED 

Human  Research  Protection  Office  IRB 

3 

The  HRPO  has  granted  exempt  status.  COMPLETED 

Milestone(s)  Achieved: 

1 .  Project  infrastructure  in  place 

2.  Local/Site  IRB  and  HRPO  Approval 

6 

100%  COMPLETED 

Task  2:  Identify  constructs  of  interest 

Literature  search  strategy 

0-3 

Search  strategy  within  healthcare  literature,  trauma 
performance  literature,  trauma  outcomes  literature,  and  team 
science  has  been  defined.  COMPLETED 

Review  of  identified  manuscripts  and 
literature 

0-6 

The  review  of  relevant  literature  (healthcare  and  team  science) 
to  inform  the  conceptual  model  and  framework  of  adaptive 
performance  has  been  completed.  COMPLETED 

Milestone(s)  Achieved: 

1 .  Identification  of  individual  and  team 
performance  constructs  for  the 
conceptual  framework  and  training 
architecture 

6 

We  identified  relevant  individual  and  team  constructs  and 
designed  a  draft  framework.  We  anticipate  continuing  to  revisit 
this  framework  as  model  testing  occurs  (Aim  2).  We  show  this 
as  an  ongoing  milestone  nearly  complete.  On  time,  99% 
completed 

Task  3:  Determine  relevant  variables  and  relationships 

Develop  nomological  net  among  constructs 
identified  in  Task  2 

3-9 

We  have  identified  key  relationships  between  processes  and 
variables  critical  for  team  adaptability.  COMPLETED 

Subject  matter  expert  review  of  variables  and 
relationships 

6-9 

Trauma  care  and  military  experts  reviewed  the  components  of 
our  adaptability  model.  Modifications  included  the  addition  of 
cognitive  adaptability  and  diagnostic  process  as  a  key 
component  of  trauma  team  adaptive  capacity.  COMPLETED 

Milestone(s)  Achieved: 

1 .  Identification  of  individual  and  team 
performance  constructs  for  the 
conceptual  framework  and  training 
architecture 

9 

100%  COMPLETED 

Task  4:  Identify  appropriate  level  of  constructs  and  variables 

Identification  of  appropriate  levels  for 
constructs,  relationships,  and  outcomes 
identified  in  Task  3 

6-9 

Literature  reviews  and  subject  matter  expert  opinion  was  used 
to  choose  and  adapt  a  model  of  individual,  team,  and  system- 
level  measurement  necessary  to  guide  the  development  and 
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implementation  of  effective  team  training.  COMPLETED. 

Milestone(s)  Achieved: 

1 .  Multilevel  framework  of  healthcare 
team  training  performance 

9 

We  identified  relevant  individual  and  team  constructs  and 
designed  a  draft  framework.  We  anticipate  continuing  to  revisit 
this  framework  as  model  testing  occurs;  therefore  will  reflect  this 
as  an  ongoing  milestone  nearly  complete.  On  time,  99% 
completed 

Task  5:  Identify  appropriate  outcome  measures  and  mechanisms 

Construct  framework  for  provision  of  adaptive 
guidance  during  simulation-based  team 
training 

6-9 

Relevant  feedback  mechanisms  and  designs  have  been 
identified  and  a  draft  framework  has  been  designed. 

COMPLETED. 

Subject  matter  expert  review  of  feedback 
framework 

9-12 

In  process.  We  currently  have  our  military,  external  team 
science,  and  external  emergency  medicine  subject  matter 
experts  reviewing  the  structure  of  our  feedback  framework  to 
ensure  the  framework  is  compatible  with  current  military  training 
efforts  and  reflective  of  current  team  science  recommendations. 
Our  military  expert  and  collaborator,  Jay  Baker,  MD,  took  a  new 
assignment.  We  have  since  identified  a  new  collaborator  from 
Madigan  Army  Medical  Center,  Capt.  Lindsay  Grubish,  DO,  to 
serve  as  a  co-investigator  on  the  project  (see  Changes  / 
Problems).  We  are  in  the  process  of  obtaining  appropriate 
approval  and  a  letter  of  support.  This  task  is  expected  to  be 
completed  within  the  next  30  -  60  days. 

Milestone(s)  Achieved: 

1 .  Integrated  team  training  design 
architecture 

2.  Evidence-based  guidelines  and 
recommendations  for  the  provision  of 
embedded,  adaptive  guidance 

12 

In  process,  90%  completed.  Anticipate  full  completion 
within  the  next  quarter. 

Task  6:  Identify  appropriate  outcome  measures  and  mechanisms 

Construct  framework  for  provision  of  adaptive 
guidance  during  simulation-based  team 
training 

6-9 

Relevant  feedback  mechanisms  and  designs  have  been 
identified  and  a  draft  framework  has  been  designed. 

COMPLETED. 

Subject  matter  expert  review  of  feedback 
framework 

9-12 

In  process.  We  currently  have  our  military,  external  team 
science,  and  external  emergency  medicine  subject  matter 
experts  reviewing  the  structure  of  our  feedback  framework  to 
ensure  the  framework  is  compatible  with  current  military  training 
efforts  and  reflective  of  current  team  science  recommendations. 
Our  military  expert  and  collaborator,  Jay  Baker,  MD,  took  a  new 
assignment.  We  have  since  identified  a  new  collaborator  from 
Madigan  Army  Medical  Center,  Capt.  Lindsay  Grubish,  DO,  to 
serve  as  a  co-investigator  on  the  project  (see  Changes  / 
Problems).  We  are  in  the  process  of  obtaining  appropriate 
approval  and  a  letter  of  support.  This  task  is  expected  to  be 
completed  within  the  next  30  -  60  days. 

Milestone(s)  Achieved: 

3.  Integrated  team  training  design 
architecture 

4.  Evidence-based  guidelines  and 
recommendations  for  the  provision  of 
embedded,  adaptive  guidance 

12 

In  process,  90%  completed.  Anticipate  full  completion 
within  the  next  quarter. 

7 


Aim  2.  Develop  and  refine  a  predictive  model  of  trauma  team  performance  and  outcomes  for  use  in 
an  adaptive  guidance/feedback  system 

The  primary  outcome  from  Aim  2  is  a  predictive  trauma  team  performance  assessment  tool  that  generalizes  to 
teams  of  varying  expertise  levels  and  across  civilian  and  military  contexts  and  is  capable  of  supporting 
embedded,  adaptive  guidance  during  simulation-based  team  training.  Our  approach  examines  the  use  of 
Bayesian  Belief  Networks  (BBNs)  to  support  the  provision  of  adaptive,  embedded  guidance  that  facilitates 
development  of  adaptive  expertise  and  trauma  team  performance.  We  utilize  existing  simulation-based  trauma 
team  performance  data  to  construct  a  BBN  that  models  the  relationships  between  key  individual  and  team 
characteristics,  behavioral  outcomes,  and  patient  care  events  in  a  previously  well-defined  and  validated 
simulated  scenario.  The  model  will  leverage  the  probabilistic  interdependencies  among  these  variables  to 
enable  educators  and/or  learners  to  assess  the  likelihood  of  critical  team/patient  outcomes  in  the  simulated 
environment.  We  then  incorporate  the  design  architecture  conceptual  foundations  developed  in  Aims  la&b  to 
guide  the  transformation  of  predictive  model  data  into  an  adaptive  guidance  tool.  The  tasks,  timeline,  and 
status  of  each  step  associated  with  Aim  2  are  summarized  in  the  table  below. 


Aim  2:  Major  Goals  and  Tasks 


SPECIFIC  AIM  2 

Timeline  . 

...  ..  .  Status 

(Months) 

Task  1 :  Collection  of  prospective  simulation  data 

Subject  recruitment 

4-6 

Completed,  100%  completed 

Execute  trauma  resuscitation  simulations 

4-6 

Due  to  resident  scheduling,  trauma  resuscitation  simulations 
began  04/2016.  We  were  unable  to  complete  all  simulations 
prior  to  the  change  in  academic  year.  We  anticipate  completion 
Oct  30,  2016. 

Delayed,  80%  completed 

Train  and  calibrate  raters 

6 

Rater  training  has  been  designed  to  code  new  simulations. 
Existing  trauma  videos  have  been  coded,  with  excellent  inter¬ 
rater  reliability.  We  anticipate  refresher  training  periodically. 

COMPLETED 

Code  videos  of  simulated  resuscitations 
using  patient  care  and  teamwork  measures 

6-12 

Simulation  video  processing  has  slightly  delayed  the  initiation  of 
coding;  coding  is  now  underway.  We  anticipate  completion  at  the 
end  of  Q6.  To  ensure  timely  completion,  we  have  hired  additional 
video  processors  and  purchased  additional  storage  to  allow 
more  rapid,  efficient  video  processing. 

Delayed,  50%  complete,  (Q5  completion  planned) 

Transform  data  into  appropriate  categorical 
structure  for  BBN 

9-12 

Using  subject  matter  input,  we  are  transforming  existing  data  into 
a  categorical  structure  to  facilitate  BBN  development.  This  is 
required  to  execute  BBN  modeling  and  requires  the  input  of 
clinical  experts.  The  overall  structure  of  the  BBN  has  been 
determined  (Figure  5).  Data  transformation  steps  have  been 
initiated.  This  process  requires  multiple  iterations  and  testing. 

This  subtask  is  slightly  delayed,  we  expect  to  be  complete  by  the 
end  of  Q5. 

Delayed,  20%  completed,  (Q5  completion  planned) 

Milestone(s)  Achieved: 

1 .  Team  data  set  of  teamwork  and  patient 
care  performance  during  trauma 
resuscitation  simulation 

12 

Delayed,  50%  completed,  (Q5  completion  planned) 

Task  2:  Identify  and  define  variables  (nodes)  for  inclusion  in  team  assessment  model 

Examination  of  conceptual  frameworks  and 
literature  review  from  Aims  la  and  1b 

9-12 

This  work  is  in  process  and  nearly  complete.  We  are  finalizing 
the  review  of  feedback  principles  to  make  final  decisions 
regarding  when  the  BBN  will  be  designed  to  provide  information 
to  learners  and  instructors  and  in  what  format  the  feedback 
should  be  delivered.  This  subtask  is  delayed  by  approximately  1 
month. 

Delayed,  80%  completed,  (Q5  completion  planned) 
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Evaluation  of  existing  experimental  dataset 
to  identify  and  extract  variables  of  interest 

9-12 

We  have  initiated  review  of  existing  datasets  for  candidate 
variables  appropriate  for  inclusion  in  the  BBN.  Transforming  the 
existing  dataset  required  more  work  than  anticipated.  This 
subtask  is  therefore  delayed.  We  anticipate  completion  no  later 
than  Q6.  As  this  step  is  occurring  simultaneously  with  Task  3,  we 
do  not  anticipate  being  more  than  3  months  delayed  overall. 
Delayed,  20%  completed,  (Q6  completion  planned) 

Milestone(s)  Achieved: 

1.  Identification  of  observable  measures 
and  latent  constructs  to  be  incorporated 
into  the  BBN 

12 

Delayed,  50%  completed,  (Q6  completion  planned) 

Task  3:  Design  the  structure  for  the  prototype  BBN 

team  assessment  system 

Identify  appropriate  and  parsimonious 
candidates  for  the  causal  structure  among 
the  variables 

12-15 

Planned,  0%  completed 

Subject  matter  expert  review  of  variable 
relationships 

12-15 

Planned,  0%  completed 

Milestone(s)  Achieved: 

1 .  Identification  of  multiple  candidate 

BBNs  for  the  observed  variables 

15 

Planned,  0%  completed 

Task  4:  Generate  initial  probability  tables  for  BBN  team  assessment  system 

Transform  data  into  appropriate  categorical 
structure 

12-15 

Planned,  0%  completed 

Explore  different  learning  algorithms 

15-18 

15-18 

Generate  conditional  dependencies  for 
unavailable  data 

15-18 

Planned,  0%  completed 

Milestone(s)  Achieved: 

1.  Functional  prototype  BBN  team 
assessment  system 

18 

Planned,  0%  completed 

Task  5:  BBN  team  assessment  system  calibration 

Transform  prospective  data  into 
appropriate  categorical  structure  for  BBN 

12-15 

Planned,  0%  completed 

Use  prospectively  collected  data  to 
calibrate  BBN 

18-21 

Planned,  0%  completed 

Use  subject  matter  experts  and  empirical 
data  from  the  literature  review  in  Aim  la  to 
adapt  the  BBN  as  needed 

18-21 

Planned,  0%  completed 

Milestone(s)  Achieved: 

1.  Functional,  generalizable  prototype 

BBN  trauma  team  assessment  system 

21 

Planned,  0%  completed 

Task  6:  Report  writing  and  dissemination 

Prepare  final  report  and  manuscripts 

21  -24 

Planned,  0%  completed 

Submit  final  reports  and  manuscripts 

24 

Milestone(s)  Achieved: 

1.  Dissemination  of  methodological 
approach  and  empiric  findings 

24 

Planned,  0%  completed 

BBN  =  Bayesian  Belief  Network 
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3b.  What  was  accomplished  under  these  goals  (organized  by  Aim)? 


Aim  la.  Develop  a  team  training  design  architecture  to  support  simulation-based  training/assessment 
systems  capable  of  developing  adaptive  expertise  in  health  care  teams 
Aim  1  b.  Develop  evidence-based  guidelines  and  recommendations  for  the  development  of  embedded, 
adaptive  feedback  and  performance  assessments 


Data  Collection:  A  robust  literature  review  is  critical  to  the  development  of  a  comprehensive  health  care  team 
training  design  architecture.  We  conducted  an  extensive  literature  review,  both  within  healthcare  and  team 
science  literature  to  identify  key  components  of  team  performance  adaptability.  We  focused  specifically  on 
identifying  the  individual  and  team  processes  that  drive  adaptive  behaviors,  as  well  as  possible  metrics  that 
would  indicate  adaptability  at  individual  and  team  levels.  We  then  convened  a  multidisciplinary  group  of  nurses 
and  physicians  from  both  civilian  and  military  health  care  settings  to  provide  expertise  and  insight  into  how 
these  adaptive  behaviors  translate  to  the  health  care  setting,  and  how  they  might  develop  over  different  levels 
of  expertise.  Finally,  we  observed  both  simulated  and  actual  trauma  team  performance  to  augment  our  data 
and  further  our  understanding  of  how  adaptive  performance  unfolds  during  highly  complex  clinical  activities. 
This  information  was  then  used  to  inform  the  identification  of  key  conceptual  models  described  below. 

Defining  Adaptive  Performance  in  Trauma  Teams:  We  used  the  literature  review  and  subject  matter  expert 
review  described  above  to  identify  all  individual  and  team-performance  concepts  and  constructs  that  are 
relevant  to  training,  assessing,  and  supporting  adaptive  trauma  team  performance.  Our  initial  adaptive 
performance  model  did  not  reflect  the  need  for  trauma  teams  to  rapidly  incorporate  new  diagnostic  information 
into  the  team’s  plans  and  processes.  Subject  matter  experts  concerned  raised  this  issue  that  cognitive 
processes  were  not  adequately  represented.  We  therefore  reviewed  the  diagnostic  error  literature,  diagnostic 
decision-making  literature,  and  team  learning  research  to  augment  our  model.  The  result  is  listed  in  Figure  1. 


Figure  1 . 
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This  model  reflects  the  cognitive  and  behavioral  process  components  of  trauma  team  performance.  First, 
cognition  is  represented  by  the  team’s  efforts  to  make  sense  of  the  situation  (Situation  Assessment).  Briefly, 
the  team  must  use  existing  data/observations  to  identify  the  patient-  and  team-related  tasks  and  demands. 

This  information  is  then  used  to  develop  a  differential  diagnosis.  Based  on  this/these  diagnoses,  the  team  has 
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expectations  regarding  how  the  patient  will  respond  to  treatments  and  how  his/her  condition  will  evolve  over 
time.  The  team  continuously  compares  this  “expected”  state  to  the  “observed”  state  of  the  patient.  This 
comparison  informs  the  team  and  helps  regulate  the  team  processes  that  regulate  task  performance.  If  the 
team  notes  a  mismatch  between  expected  patient  improvement  and  current  patient  condition,  this  should 
prompt  the  team  to  review  their  plan,  make  adjustments,  and  execute  the  modified  plan.  The  results  of  these 
new  actions  should  be  monitored  and  evaluated.  The  observations  made  during  evaluation  become  the 
information  that  the  team  uses  to  reassess  the  situation,  reconsider  the  differential  diagnosis(es),  and  the 
adaptive  cycle  continues.  In  a  rapidly  evolving  trauma  resuscitation,  this  cycle  repeats  continuously  to  ensure 
the  team  is  adapting  to  the  unstable  patient/team/environment. 

Identifying  appropriate  training  targets:  Training  should  be  purposeful  and  should  target  appropriate 
cognitive,  behavioral,  and  affective/motivational  processes  in  a  stepwise  fashion.  Training  mechanisms  should 
support  both  skill  implementation  in  the  clinical  environment  as  well  as  transfer  to  novel  situations.  We 
identified  a  staged  approach  to  training  that  targets  appropriate  skills  necessary  to  develop  adaptive  capacity. 
We  include  both  individual  and  team-based  processes  as  well  as  training  mechanisms.  The  framework  below 
(Figure  2)  provides  an  outline  for  this  approach. 


Figure  2.  Training  targets  and  training  techniques 
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Identifying  appropriate  level  of  constructs  and  variables:  A  thorough  understanding  of  individual  and  team 
performance  within  complex  environments  necessitates  a  multilevel  approach  to  theory-building  and  outcomes 
research.  Organization-level  phenomena  emerge  through  the  behavior,  perceptions,  affect,  and  interactions  of 
individuals  and  team.  Likewise,  individuals  and  teams  are  directly  influenced  by  the  culture,  norms,  and 
structure  of  the  organization.  Ignoring  the  multilevel  nature  of  a  construct,  intervention,  or  relationship  may 
result  in  oversimplification  of  outcomes  and  failure  to  recognize  important  measurement  targets.  We 
developed  a  multilevel  conceptual  architecture  of  adaptation  that  considers  (1)  the  types  of  events  teams  must 
adapt  to  (i.e.,  what  type  of  change  is  occurring),  (2)  the  types  of  processes  teams  use  to  adapt,  and  (3)  at  what 
level  these  processes  occur.  This  taxonomy  (Figure  3)  can  help  guide  the  selection  of  appropriate  training 
targets  and  can  help  educators  target  correct  task  complexity,  appropriate  processes 
(cognitive/behavioral/affective),  and  direct  training  and  measurement  at  the  correct  level  (individual,  team, 
unit).  Such  specificity  is  important,  as  being  purposeful  when  designing  training  will  ensure  that  individuals, 
teams,  and  units  are  prepared  for  the  specific  types  of  adaptation  necessary  for  their  work.  This  level  of 
specificity  in  training  is  often  overlooked  and  is  not  part  of  current  training  guidelines.  In  Appendix  1  we 
describe  training  principles  related  to  (1)  level  of  training  and  (2)  specific  processes  targeted  by  training.  In 
Appendix  2  we  then  describe  three  different  task  requirements  for  adaptability  and  specifically  identify  training 
principles  associated  with  each  type  of  task  complexity. 
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Figure  3.  Model  of  task  complexity,  processes,  and  level(s)  of  analysis 
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Identifying  appropriate  outcome  measures  and  mechanisms:  We  noted  that  training  evaluation  systems 
should  consider  both  proximal  and  distal  outcomes.  Proximal  outcomes  include  both  learning  and 
performance-based  outcomes  and  can  include  basic  declarative  knowledge  as  well  as  more  complex  strategic 
knowledge  and  performance.  Distal  outcomes  that  are  trainee-focused  include  the  transfer  of  learned  skills  to 
the  work  (clinical)  environment  as  well  as  the  application  of  learned  skills  to  novel  situations,  i.e.,  adaptability. 
High-level  distal  outcomes  include  patient,  system,  and  organization-level  outcomes.  Our  literature  review 
focused  on  the  identification  of  pertinent  proximal  and  distal  outcomes.  We  considered  our  own  systematic 
reviews  as  well  as  other  health  care  team  reviews  to  determine  the  current  state  of  team  assessment.  We 
extended  this  knowledge  by  investigating  the  team  science,  safety  science,  and  human  factors  literature. 
Because  our  work  focuses  on  developing  adaptive  expertise,  considerable  efforts  were  made  to  identify 
outcome  measures  that  reflect  adaptive  capacity.  Subject  matter  expert  review  was  utilized  to  help  identify 
where  non-health  care  team  assessments  can  be  translated  into  appropriate  health  care  team  training 
evaluation  targets.  In  Figure  4  we  propose  a  translational  simulation-based  research  model  that  considers 
appropriate  outcome  measures  and  relationships  for  individual  and  team-level  adaptability. 


Figure  4.  Multilevel  outcome  model  for  training  evaluation 
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Recommendations  for  the  provision  of  adaptive  feedback:  For  the  purposes  of  this  work,  we  considered 
(1)  performance  measures  used  for  the  provision  of  feedback  and  (2)  training  evaluation/outcome  metrics  used 
to  measure  training  impact,  separately.  The  provision  of  feedback  is  a  major  focus  of  this  study,  with  the  goal 
of  developing  an  assessment  system  capable  of  supporting  embedded,  adaptive  guidance.  We  therefore 
directed  our  efforts  towards  developing  a  conceptual  framework  to  support  the  content,  structure,  and  provision 
of  adaptive  guidance  during  trauma  team  simulations.  This  work  relied  heavily  on  the  training,  education,  and 
debriefing  literatures.  In  Appendix  3  we  list  feedback  principles,  scientific  rationale,  and,  where  appropriate, 
exemplars  for  simulation-based  training. 

Deliverables:  We  plan  a  formal  report  of  final  recommendations  and  findings  from  Aims  la  &  1b  to  be 
disseminated  no  later  than  Q6.  These  results  will  comprise  the  material  disseminated  over  the  next  year. 

Aim  2.  Develop  and  refine  a  predictive  model  of  trauma  team  performance  and  outcomes  for  use  in  an 
adaptive  guidance/feedback  system 

Trauma  Simulations  and  Performance  Coding:  The  purpose  of  conducting  trauma  team  simulations  is  to 
provide  baseline  data  for  the  design  of  the  BBN.  These  simulations  will  be  used,  along  with  existing  simulation 
data,  to  inform  the  structure  of  the  BBN.  Subject  recruitment  for  prospective  trauma  simulations  is  completed, 
and  the  simulation  sessions  are  underway.  We  have  identified  and  recruited  both  civilian  and  military  trauma 
team  leaders.  To  ensure  adequate  military  representation,  simulations  will  continue  until  military  trauma  team 
leaders  have  completed  the  exercise.  To  date  we  have  conducted  16  of  the  initially  proposed  20  simulations 
and  plan  to  extend  this  number  to  22  to  ensure  adequate  military  representation.  Simulation  completion  is 
slightly  delayed  but  will  be  completed  in  Q5.  As  noted  on  our  last  report,  we  are  able  to  make  use  of  existing 
trauma  simulations,  which  have  been  coded.  Rater  training  will  begin  in  the  next  quarter.  New  simulations  will 
be  coded  on  an  ongoing  basis  throughout  the  next  project  year. 


BBN  Modeling:  We  explored  several  candidate  approaches  to  BBN  design.  The  preliminary  approach 
determined  to  be  most  informative  for  the  purposes  of  the  project  is  provided  in  Figure  5.  Briefly,  this  approach 
allows  for  the  identification  of  critical  feedback  on  key  team  processes  (e.g.,  Mission  Analysis  [MA];  Goal 
Specification  [GS];  Strategy  Formulation  [SF];  Systems  Monitoring  [SM])  at  set  points  throughout  the 
simulation.  This  type  of  feedback  encourages  individuals  and  teams  to  consider  contingency  planning  and  to 
actively  evaluate  their  performance  and  make  real-time  adjustments  as  needed,  i.e.,  adapt. 


Figure  5.  Proposed  BBN  approach 
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BBN  Variables:  V\le  initiated  review  of 
existing  datasets  for  candidate  variables 
appropriate  for  inclusion  in  the  BBN. 
Variables  are  considered  appropriate  if 
there  is  variability  amongst  subjects,  and  if 
variables  correlate  with  overall 
performance  and  process  as  a  whole. 

This  requires  evaluating  over  100  process 
variables  and  80  performance  variables. 
With  the  input  of  subject  matter  experts, 
we  are  vetting  the  appropriateness  of 
variables  and  are  considering  grouping 
certain  variables  into  composite  indicators 
for  inclusion  in  the  BBN.  This  would 
potentially  simplify  BBN  input  during 
testing  and  refinement.  Finally,  subject 
matter  experts  are  considering  variable 
states  and  relationships  to  ensure  the 
BBN  structure  accurately  reflects  civilian 
and  military  trauma  care.  Candidate 
variables  are  listed  in  Appendix  4. 
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3c.  What  opportunities  for  training  and  professional  development  has  the  project  provided? 

Subjects  enrolled  in  the  study  received  simulation-based  trauma  team  training  and  assessments.  The 
simulation  overview  is  provided  in  Appendix  C  as  noted  above.  While  the  provision  of  training  is  not  a  major 
focus  of  this  project,  trainees  were  able  to  practice  trauma  management  skills  as  well  as  leadership  skills  under 
difficult  conditions  requiring  significant  individual  and  team  adaptation. 

3d.  How  were  the  results  disseminated  to  communities  of  interest? 

Dissemination  of  our  work  from  Aims  la  and  1b  is  planned  for  Q5  and  Q6.  Specifically,  we  are  preparing  two 
manuscripts,  one  describing  our  frameworks,  training  principles,  and  concepts  related  to  adaptability  and  a 
second  related  to  the  provision  of  adaptive  feedback.  Adaptive  feedback  is  a  relatively  new  concept  within 
medical  simulation  and  one  that  needs  to  be  considered  within  the  growing  literature  around  debriefing  and  the 
provision  of  performance-related  information.  We  will  also  submit  this  work  in  the  upcoming  year  to  the  Military 
Health  System  Research  Symposium,  the  Annual  Meeting  for  the  Society  for  Academic  Emergency  Medicine, 
and  the  Annual  Meeting  for  the  Human  Factors  and  Ergonomics  Society.  This  will  ensure  wide  distribution  of 
information  in  military  and  civilian  healthcare  arenas  as  well  as  within  the  training  and  human  factors 
community. 

3e.  What  do  you  plan  to  do  during  the  next  reporting  period  to  accomplish  the  goals? 

Aim  la.  Develop  a  team  training  design  architecture  to  support  simulation-based  training/assessment 
systems  capable  of  developing  adaptive  expertise  in  health  care  teams 
Aim  1  b.  Develop  evidence-based  guidelines  and  recommendations  for  the  development  of  embedded, 
adaptive  feedback  and  performance  assessments 

Integrate  Conceptual  Frameworks,  Evidence-based  Training  and  Assessment  Mechanisms:  Work  for 
Aims  la  and  1b  will  focus  primarily  on  integration  of  the  models  and  dissemination.  Using  an  approach  outlined 
by  Rousseau,  et  al.  the  investigators  will  integrate  the  information  obtained  in  Steps  1  through  4  (individual- 
and  team-level  exogenous  variables,  critical  teamwork  processes  and  emergent  states,  multilevel 
relationships,  and  important  proximal  and  distal  outcome  measurement  targets)  into  a  comprehensive  team¬ 
training  design  architecture.  We  will  use  the  literature  review  to  identify  knowledge  gaps  and  immediate 
research  targets.  Specific  attention  will  be  given  to  important  variables,  implementation  factors,  and  measures 
associated  with  the  provision  of  embedded,  adaptive  guidance.  The  products  of  this  work  will  be: 

Deliverable  1.  Health  Care  Team  Training  Design  Architecture.  A  unified,  evidence-based  conceptual 

framework  of  health  care  team  training  effectiveness  that  identifies  critical  variables  -  individual 
and  team  factors,  training  design  elements,  and  training  implementation  methods  -  that  can  be 
leveraged  to  improve  team  adaptive  expertise  and  performance  through  robust  simulation- 
based  training  systems 

Deliverable  2.  Embedded,  Adaptive  Guidance:  Guidelines  and  Recommendations.  Clear  guidelines  and 
recommendations  for  the  design,  development,  and  implementation  of  embedded,  adaptive 
guidance  to  optimize  team  adaptability  and  team  performance 

These  deliverables  will  be  disseminated  in  two  manuscripts.  Manuscript  preparation  and  submission  is  planned 
for  Q5  and  Q6  during  project  year  2. 


Aim  2.  Develop  and  refine  a  predictive  model  of  trauma  team  performance  and  outcomes  for  use  in  an 
adaptive  guidance/feedback  system 

Identification  and  Definition  of  Variables  (Nodes)  for  Inclusion  in  Team  Assessment  Model:  The  first 
step  in  constructing  any  team  assessment  system  involves  defining  the  focal  constructs  of  interest  to  be 
captured  and  specifying  how  they  will  be  operationalized.  However,  assessment  system  designers  must  also 
recognize  practical  considerations  (e.g.,  purpose  of  measurement,  feasibility  of  implementation/use,  etc.)  when 
deciding  what  should  be  assessed  and  how  observations  will  be  captured.  The  investigators’  are  using  their 
existing  research  and  dataset  to  identify  and  extract  variables/indicators  that  can  be  used  to  capture  these 
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constructs.  Subject  matter  experts  in  both  team  performance  and  health  care  practice  have  been  recruited  to 
review,  supplement,  and  validate  the  final  assessment  targets.  The  result  of  this  step  will  be  identification  of  the 
directly  observable  measures  and  latent  constructs  that  will  be  incorporated  into  the  prototype  BBN  team 
assessment  system.  This  work  is  currently  in  process  and  will  be  completed  by  Q6. 

Designing  the  Structure  for  the  Prototype  BBN  Team  Assessment  System:  An  important  consideration 
when  designing  a  BBN  is  adequate  specification  of  the  arcs  and  their  probabilities.  Depending  on  the  goals  of 
the  predictive  tool,  a  given  set  of  nodes  in  a  BBN  may  be  linked  together  in  different  ways.  Although  these 
different  configurations  do  not  change  the  nature  of  the  joint  probability  space  modeled  by  the  BBN,  they  do 
hold  implications  for  how  probability  propagation  occurs  and  inferences  are  drawn,  which  subsequently 
impacts  how  the  model  can  be  implemented  in  practice.  Given  the  goals  of  the  present  project  to  leverage  the 
predictive  capabilities  of  BBNs  as  an  assessment  platform  to  facilitate  real-time  adaptive  guidance,  this  step 
involves  identifying  appropriate  and  parsimonious  candidates  of  the  causal  structure  among  the 
variables/indicators  identified  for  inclusion  in  the  model.  The  purpose  of  the  prototype  BBN  model  in  this  project 
is  to  serve  as  an  assessment  tool  that  could  be  used  in  a  predictive  fashion  to  facilitate  adaptive  guidance  and 
expertise.  As  such,  the  complexity  and  precision  of  the  influence  network  must  be  balanced  so  as  to  be  both 
informative/evidence-based  (i.e.,  critical  relationships  are  modeled)  yet  feasible  to  implement  real-time  (i.e., 
minimally  sufficient  number  of  indicators  to  monitor).  The  investigators  will  once  again  rely  on  multiple 
resources  to  inform  these  decisions.  The  final  result  of  this  step  will  be  identification  of  multiple  candidate 
BBNs  for  the  observed  variables  that  could  serve  as  potential  predictive  assessment  models  in  the  present 
learning  environment. 

Generation  of  Initial  Probability  Tables  for  BBN  Team  Assessment  System:  The  computational  “engine” 
and  predictive  validity  of  a  BBN  relies  on  the  presence  of  well-informed  conditional  probability  tables  (CPTs).  A 
CPT  exists  for  every  node  in  a  BBN  and  reflects  the  probability  that  a  particular  state  for  a  particular  node  will 
be  observed  given  the  state  of  all  its  parent  nodes  (e.g.,  p(Chest  Compression  Quality  =  High  |  Assign  a  Team 
Leader  =  No),  etc.).  In  this  sense,  CPTs  represent  the  degree  of  interdependency  (i.e.,  correlation)  that  exists 
between  variables  that  share  a  directed  arc.  To  compute  the  CPTs  for  the  candidate  networks,  the 
investigators  will  utilize  their  existing  dataset  to  “train”  a  set  of  initial  conditional  probabilities  for  the  modeled 
variables.  This  process  will  entail  several  steps.  First,  data  must  be  transformed  into  an  appropriate  categorical 
structure  that  can  be  interpreted  by  a  BBN.  Next,  different  learning  algorithms  will  be  explored  (i.e.,  counting, 
expectation-maximization,  gradient  descent)  in  an  attempt  to  produce  the  “maximum  likelihood  BBN,”  or  the  set 
of  CPTs  that  is  most  likely  given  the  observed  data.  The  fit  of  the  algorithms  will  be  assessed  using  standard 
model  evaluation  techniques  (e.g.,  confusion  matrix,  times  surprised,  etc.);  additionally,  these  metrics  will  be 
used  to  compare  candidate  BBNs  to  identify  the  best  fitting  model.  Finally,  in  instances  where  data  is 
unavailable  or  insufficient  to  generate  a  suitable  CPT,  existing  empirical  literature  (i.e.,  meta-analyses)  and/or 
subject  matter  experts  will  be  relied  upon  to  generate  the  nature  of  the  conditional  dependence.  The  result  of 
this  step  will  be  development  of  the  best  fitting,  functional  prototype  BBN  team  assessment  system  based  on 
existing  data. 

BBN  Team  Assessment  System  Calibration:  A  potential  concern  with  using  only  a  single  sample  to 
construct  a  BBN  is  that  the  model  and  its  accompanying  CPTs  may  be  “overfit”  and  fail  to  generalize  beyond 
the  training  data.  Thus,  in  the  final  step  of  development,  the  performance  of  the  BBN  team  assessment  system 
will  be  evaluated  and  recalibrated  using  the  new  data  collected  through  coding  of  simulations.  A  similar 
approach  to  evaluating  model  fit  as  described  above  will  be  implemented  to  examine  the  adequacy  of  the 
BBN’s  predictions  in  the  new  data.  To  the  extent  that  misfits  among  particular  nodes  or  relationships  are 
identified,  the  investigators  will  rely  on  subject  matter  experts  and  empirical  evidence  from  the  literature  to 
identify  whether  and/or  how  to  adapt  the  BBN  (adjust  CPTs,  specify  new  nodes/variables,  revise  causal 
pathways).  Irrespective  of  fit,  the  new  data  can  be  used  to  improve  the  precision  of  the  BBN  assessment  model 
through  added  observations.  The  results  of  this  step  will  thus  be  improvement  and  calibration  of  the  prototype 
BBN  team  assessment  system. 
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4.  Impact  for  Project  Year  1  Work 

4a.  What  was  the  impact  on  the  development  of  the  principal  discipline(s)  of  the  project? 

Our  work  will  improve  training,  maximize  healthcare  provider  performance,  and  minimize  morbidity  for  our 
injured  service  men  and  women.  Once  disseminated,  the  work  from  project  year  one  will  provide  military  and 
civilian  healthcare  providers  and  educators  with  clear  guidelines  for  the  development  of  training  that  builds 
adaptive  capacity.  Specifically,  we  provide  developmental^  appropriate  training  targets  for  individuals  and 
teams.  We  identify  what  training  content  and  delivery  method  is  most  appropriate  for  developing  adaptive 
behaviors  around  certain  types  of  tasks.  We  recognize  that  frontline  medics  adapt  to  different  situations  than 
physicians  in  specialty  clinics  and  our  guidelines  account  for  these  differences.  We  aim  to  provide  a  clear, 
easily  applied  method  to  help  educators  and  trainers  make  decisions  regarding  training  development  and 
implementation.  Our  work  will  facilitate  the  development  of  longitudinal  curricula  across  multiple  specialties  and 
disciplines  by  providing  clear  training  targets  for  individuals  and  teams  at  all  levels  of  performance. 

The  guidelines  and  principles  for  adaptive  feedback  introduce  a  new  and  important  concept  to  healthcare. 
The  provision  of  “feedback”  and  “debriefing”  in  experiential  training  has  been  identified  as  critical  to  learning. 
However,  the  role  for  adaptive  feedback  in  the  development  of  highly  adaptive  teams  has  not  been  described. 
We  will  disseminate  our  review  of  the  topic  along  with  specific  recommendations  for  implementation  within 
simulation-based  training.  Along  with  the  work  to  be  performed  in  Aim  2,  this  information  will  provide  the 
foundation  for  the  development  of  simulation-based  training  with  automated,  adaptive  feedback. 

4b.  What  was  the  impact  on  other  disciplines 

Our  work  has  impact  beyond  healthcare.  We  highlight  the  challenges  associated  with  training  and  evaluating 
performance  in  complex  environments.  This  information  is  useful  in  human  factors  and  organizational 
psychology,  where  teamwork  has  often  been  considered  a  static  construct,  rather  than  a  dynamic  entity  where 
teams  learn,  adapt,  and  react  to  continuous  changes  in  the  task,  environment,  and  team.  Our  framework 
highlights  how  important  it  is  to  consider  characteristics  of  the  task(s)  necessitating  adaptation  when 
developing  training  programs.  This  work  provides  a  foundation  to  build  more  comprehensive  training  that  goes 
beyond  TeamSTEPPS-type  training  to  impact  complex  teams  performing  in  highly  dynamic,  potentially 
dangerous  situations. 

4c.  What  was  the  impact  on  technology  transfer? 

Nothing  to  report 

4d.  What  was  the  impact  on  society  beyond  science  and  technology? 

Failure  to  adapt  to  rapidly  changing  conditions  is  a  primary  cause  of  medical  error.  In  military  settings,  such 
failures  can  also  lead  to  significant  harm  to  providers.  Our  work  has  a  significant  impact  on  patient  safety, 
decreasing  soldier  morbidity  and  mortality,  and  on  patient  satisfaction.  Simulation  is  a  key  modality  leveraged 
by  the  military  to  advance  expertise  and  ensure  that  soldiers  receive  the  highest  level  of  clinical  care. 
Significant  human  and  technological  resources  are  dedicated  to  developing  and  implementing  rigorously 
tested,  high-quality  simulation-based  curricula.  Clear  guidelines  and  a  training  framework  focused  on 
developing  adaptive  capacity  did  not  exist.  We  fill  this  gap  and,  in  doing  so,  provide  an  important  mechanism 
to  support  the  development  and  implementation  of  highly  effective  individual  and  team-level  healthcare 
training. 
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5.  Changes  /  Problems 


5a.  Changes  in  approach  and  reasons  for  change 

None 

5b.  Actual  or  anticipated  problems  or  delays  and  actions  or  plans  to  resolve  them 

Scheduling  the  simulations  was  slightly  delayed  due  to  the  residents’  (subject)  clinical  schedules.  Simulations 
are  now  underway.  To  ensure  adequate  military  representation,  we  will  continue  executing  simulations  through 
the  beginning  of  PY2  to  obtain  sufficient  military  subject  data.  We  do  not  anticipate  any  additional  delays. 

We  anticipated  completing  simulation  coding  at  the  end  of  the  first  project  year.  This  work  was  slightly  delayed 
due  to  the  need  to  train  new  coders.  To  assist  with  this  work,  the  graduate  student  (Santoro)  at  Michigan  State 
University  will  be  focusing  on  monitoring  and  re-training  coders  as  needed.  This  will  also  assist  with  the 
additional  work  associated  with  coding  additional  simulations  planned  for  the  study.  We  anticipate  this  will 
address  the  problem  and  will  do  so  without  impacting  our  budget.  Ms.  Santoro  will  remain  on  the  project  until 
its  completion.  Additionally,  we  are  hiring  a  second  research  coordinator  (0.5  FTE)  to  assist  with  both 
simulations  and,  more  importantly,  formatting  coded  performance  data  to  ensure  it  can  be  quickly  used  for  the 
BBN  work  in  Aim  2.  We  had  hoped  to  have  this  individual  identified  at  this  point.  S/he  will  start  on  January  1 
and  a  CV  will  be  forwarded  when  available. 

In  summary,  we  have  experienced  slight  but  manageable  delays  that  will  not  impact  the  completion  of  the 
project  within  the  proposed  budget.  Our  solutions  are  in  process  and  we  anticipate  no  new  issues. 

5c.  Changes  that  had  a  significant  impact  on  expenditures 

The  project  is  currently  on  budget.  Delays  in  hiring  the  research  assistant  and  delay  in  starting  simulations 
have  shifted  some  of  the  costs  from  project  year  1  to  project  year  2.  Subcontract  costs  are  encumbered  now 
for  years  1  and  2.  The  slight  delays  described  above  do  not  impact  the  budget,  and  we  fully  anticipate 
completing  the  project  within  the  proposed  budget. 

5d.  Significant  changes  in  use  or  care  of  human  subjects,  vertebrate  animals,  biohazards,  and/or  select 
agents 

None.  While  we  increased  our  enrollment  to  22  teams,  this  protocol  is  exempt  and  no  further  action  is  needed. 
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6.  Products 


6a.  Publications,  conference  papers,  and  presentations 

Fernandez  R,  Rosenman  ER;  Santoro  J,  Pacic  E,  Golden  SJ,  Brolliar  SM,  Chao  GT,  Grand  JA,  Kozlowski 
SWJ.  A  multicenter,  observational  study  of  teamwork,  team  cognition,  and  leadership.  2016  Military  Health 
System  Research  Symposium ,  Orlando,  FL. 

6b.  Website  or  other  Internet  sites 

None 

6c.  Technologies  or  techniques 

None 

6d.  Inventions,  patent  applications,  and/or  licenses 

None 

6e.  Other  products 

None 
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7.  Participants  &  Other  Collaborating  Organizations 


7a.  What  individuals  have  worked  on  the  project? 


Name: 

Project  role: 
era  Commons  ID: 

Nearest  person  month  worked: 
Contribution  to  project: 


Rosemarie  Fernandez,  MD 
Principal  Investigator 
av9546 

2  cal. months  (0.2  FTE) 

Worked  with  ER  to  review  actual  trauma  resuscitations  and  identify 
missing  components  of  the  conceptual  framework.  Worked  with  JG,  ER, 
GC  to  modify  the  conceptual  framework  to  include  a  cognitive 
component.  Recruited  subjects  for  simulation  and  began  simulation 
implementation. 


Name: 

Project  role: 
era  Commons  ID: 

Nearest  person  month  worked: 
Contribution  to  project: 


James  Grand,  PhD 
Co-Principal  Investigator 
Grandjam 

3  cal.  Months  (0.25  FTE) 

Worked  with  GC  to  conduct  team  science  component  of  literature  review. 
Worked  with  RF,  ER,  GC  to  modify  conceptual  framework.  Identified 
team  science-related  training  principles  and  recommendations. 


Name: 

Project  role: 

Nearest  person  month  worked: 
Contribution  to  project: 


Elizabeth  Rosenman,  MD 

Co-Investigator 

2  cal.  Months  (0.2  FTE) 

Worked  with  RF  to  review  actual  trauma  resuscitations  and  identify 
missing  components  of  the  conceptual  framework.  Worked  with  JG,  RF, 
GC  to  modify  the  conceptual  framework  to  include  a  cognitive 
component.  Recruited  subjects  and  began  simulation  implementation. 


Name: 

Project  role: 

Nearest  person  month  worked: 
Contribution  to  project: 


Georgia  Chao,  PhD 

Co-Investigator 

2  cal.  Months  (0.19  FTE) 

Worked  with  JG  to  conduct  team  science  component  of  literature  review. 
Worked  with  RF,  ER,  JG  to  modify  conceptual  framework.  Identified  team 
science-related  training  principles  and  recommendations. 


Name: 

Project  role: 

Nearest  person  month  worked: 
Contribution  to  project: 


Karlee  Jackson,  BA 
Research  assistant 
6  cal.  Months  (0.5  FTE) 

Coordinated  subject  recruitment,  worked  with  simulation  center  to 
schedule  and  execute  simulations.  Trained  all  confederate  roles  to 
ensure  consistency  in  simulation  execution.  Worked  with  investigators  to 
develop  data  storage  solution. 


Name: 

Project  role: 

Nearest  person  month  worked: 
Contribution  to  project: 


Benjamin  Levine,  BA 

Graduate  student  research  assistant 

6  cal.  Months  (0.5  FTE) 

Performed  literature  review,  assisted  with  framework  development  and 
revision.  Worked  with  team  to  develop  BBN  approach  and  is  leading 
evaluation  of  potential  variables  for  model  inclusion. 


Name: 

Project  role: 

Nearest  person  month  worked: 
Contribution  to  project: 


Jessica  Santoro,  MA 

Graduate  student  research  assistant 

1  cal.  Months  (will  be  at  0.5  FTE  for  upcoming  project  year) 

Assisted  with  development  of  adaptive  feedback  principles,  performed 
related  literature  review,  will  be  assisting  with  rater  evaluation  and 
retraining  as  needed. 
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7b.  Has  there  been  a  change  in  the  active  other  support  of  the  PD/PIs  or  senior/key  personnel  since  the 
last  reporting  period? 

Ms.  Santoro  began  in  her  role  as  the  graduate  student  at  our  collaborating  site,  Michigan  State  University.  Ms. 
Santoro  assisted  with  the  large  literature  search  for  Aim  1  and  will  have  a  significant  role  assisting  with  rater 
evaluation  and  training  throughout  the  project. 

As  we  noted  in  our  last  report,  we  are  hiring  an  individual  to  assist  with  research  assistant  tasks.  This  person’s 
hire  was  slightly  delayed,  and  we  anticipate  them  starting  in  the  next  quarter.  This  will  help  to  facilitate  the 
remainder  of  the  simulations  and  performance  coding. 

We  intend  to  add  CPT.  Lindsay  K.  Grubish,  DO  to  the  project,  as  Dr.  Jay  Baker  took  a  new  position  and  is  no 
longer  able  to  assist  with  the  project.  CPT.  Grubish  is  a  staff  physician  in  the  Department  of  Emergency 
Medicine  at  MAMC.  She  has  experience  using  simulation  to  assess  performance  in  military  medical  providers 
under  stress.  She  will  be  responsible  for  providing  subject  matter  expertise  and  recruitment  of  military  residents 
for  the  simulations.  We  are  in  the  process  of  obtaining  appropriate  letters  of  support  and  will  provide  a  CV  and 
copy  of  letters  when  available. 

7c.  What  other  organizations  were  involved  as  partners? 


University  of  Maryland 

Department  of  Psychology 
College  Park,  Maryland 

The  Co-PI,  Dr.  Grand,  and  a  graduate  student,  Mr. 
Benjamin  Levine,  are  both  supported  at  the  University 
of  Maryland.  There,  they  have  office  space,  computer 
access,  and  support  for  virtual  meetings  with  the 
research  team. 


Eli  Broad  College  of  Business  /  Michigan  State 
University 

East  Lansing,  Michigan 

Dr.  Chao  (collaborator)  and  a  graduate  student,  Ms. 
Jessica  Santoro,  are  both  supported  at  Michigan 
State  University.  There,  they  have  office  space, 
computer  access,  and  support  for  virtual  meetings 
with  the  research  team. 


8.  Special  Reporting  Requirements 

8a.  Collaborative  Awards 

N/A 

8b.  Quad  Chart 

Please  see  Appendix  5  for  updated  Quad  Chart. 
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APPENDICES 

Appendix  1.  Training  principles  to  target  adaptive  processes  at  different  levels 
Appendix  2.  Training  principles  related  to  task  type  and  complexity 
Appendix  3.  Principles  of  providing  adaptive  feedback 
Appendix  4.  Examples  of  candidate  variables  for  the  BBN  model 
Appendix  5.  Project  QUAD  Chart 


21 


Appendix  1.  Training  principles  to  target  adaptive  processes  at  different  levels. 


Principle  and  Applicable 

Level(s) 

Rationale 

Simulation  application 

Use  advance  organizers  at  the 
start  of  training. 

(Individual  Level) 

Advance  organizers  are  materials  presented  at  the  start  of 
training  that  provide  an  initial  organizing  structure  of  the  subject 
matter  discussed  in  training.  Advance  organizers  are  used  to 
organize  conceptual  information  and  foster  connections  between 
similar  ideas  as  well  as  delineate  different  concepts  from  one 
another.  Trainees  who  use  or  develop  their  own  advance 
organizers  are  more  likely  adaptively  transfer  knowledge  and 
skills  than  those  who  did  not  use  or  develop  advance 
organizers. 

•  Inform  trainees  about  training  focus.  This  does  not 
necessarily  mean  informing  them  of  key  critical  content 
planned  for  simulations;  rather,  tell  trainees  they  will  be 
focusing  on  team  (or  individual)  skills 

•  Suggest  that  trainees  consider  personal  strengths  and 
weaknesses  prior  to  coming  to  training. 

Promote  trainees  to  have  a 
learning  goal  orientation  during 
training. 

(Individual  and  Team  Level) 

Training  design  that  promotes  a  learning  goal  orientation  (e.g.,  a 
focus  on  self-improvement  and  task  mastery  in  achievement 
situations)  has  been  linked  to  positive  training  outcomes,  such 
as  goal  setting,  self-regulatory  activities,  learning,  and 
performance.  This  is  in  stark  contrast  to  promoting  a 
performance  goal  orientation  (e.g.,  a  focus  on  demonstrating 
ability  to  others  in  achievement  situations)  which  has  been 
shown  to  negatively  relate  to  goal  striving  processes  and 
performance. 

•  Promote  a  learning  goal  orientation  by  encouraging 
trainees  to  set  goals  about  achieving  learning  objectives 
and  acquiring  relevant  knowledge  and  skills. 

•  Establish  psychological  safety 

Trainees  should  be  provided  with 
strategy  instruction  later  in  training 
once  appropriate  foundational 
knowledge  has  been  developed. 
(Individual) 

The  KSAs  required  to  effectively  engage  in  individual  and  team 
adaptation  are  advanced  learning  outcomes.  Without  achieving 
proficiency  in  the  basic  and  procedural  knowledge  necessary  to 
carry  out  core  task/job  requirements  in  a  domain,  efforts  to 
improve  the  adaptation  process  will  be  less  effective. 

•  Assess  individuals  for  team-based  simulation 
"readiness" 

•  Use  low  fidelity  non-clinical  simulations  to  begin  building 
team  skills  while  individuals  are  still  developing  clinical 
knowledge. 

•  At  this  stage,  interdisciplinary  training  is  not  important; 
however  institutions  should  ensure  consistency  of 
curriculum  across  professions/units/schools 

Training  material  should  be 
structured  so  that  instruction 
proceeds  from  general  to  detailed, 
specific  to  complex. 

(Individual  and  Team  Level) 

Successful  team  adaptation  requires  integrating,  coordinating, 
and  regulating  a  variety  of  different  KSAs,  resources,  and 
members.  Developing  the  capacities  to  manage  these 
processes  should  be  scaffolded  to  allow  learners  to  first  build 
basic  competencies  and  then  practice/engage  in  more  complex 
applications.  Note  that  this  also  applies  to  actively  training 
members  as  part  of  intact  teams  —  team-based  training 
designed  to  enhance  adaptability  is  a  complex  environment  and 
should  be  postponed  until  learners  have  engaged  in  more 
foundational  training  exercises. 

•  Team-based  simulations  should  initially  use  basic  clinical 
scenarios  rather  than  unusual  or  highly  complex 
situations.  Once  basic  team  skills  have  transferred  from 
"non-clinical"  simulations  (above)  to  straightforward 
clinical  issues,  more  complex  team  and  environmental 
issues  can  be  added. 

•  Use  EBAT  to  create  a  simulation  experience  where 
modules  can  be  added  to  model  more  complexity  as  well 
as  to  target  specific  team  skills. 
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Trainees  learning  a  complex  task 
should  be  encouraged  to  monitor 
rate  of  learning  progress  rather 
than  just  learning  performance. 
(Individual  Level) 

Training  that  emphasizes  learning  trajectories,  development, 
and  velocity  is  more  likely  to  minimize  goal  abandonment, 
promote  self-efficacy,  and  encourage  trainees  to  view  training  as 
"learning"  rather  than  "evaluation."  Additionally,  emphasizing 
"future-focused"  cognitive  appraisals  (i.e.,  focusing  on  how 
learning  outcomes/capabilities  are  evolving)  reinforces  the 
cognitive  appraisal  frames  critical  to  team  adaptation. 

•  During  pre-brief,  make  it  clear  to  learners  that  there  may 
be  no  "right  answer". 

•  Establish  a  learning  environment  that  supports 
psychological  safety. 

•  If  using  a  modular  EBAT  approach,  consider  guiding 
teams  to  recognize  how  similar  problems  were 
addressed  in  the  past  so  they  can  monitor  their 
progress. 

Trainees  learning  complex  tasks 
should  be  provided  with  proximal 
subgoals  that  break  the  task  into 
smaller  parts. 

(Individual  and  Team  Level) 

Team  adaptation  is  a  process  characterized  by  an  ongoing  cycle 
of  situation  assessment  and  team/task  management.  The  KSAs 
which  underlie  successfully  execution  of  these  stages  can  be 
developed  through  "part-learning"  and  by  breaking  the 
adaptation  process  into  meaningful  chunks.  This  approach  is 
more  likely  to  increase  learner  self-efficacy  and  persistence,  and 
allow  practice  opportunities  &  feedback  to  be  tailored  towards 
more  focused  learning  objectives. 

•  Break  down  adaptive  behaviors  into  clear  activities  that 
can  be  practiced  in  isolation.  If  necessary,  remove 
learners  from  the  clinical  setting  to  work  on  key  activities 
prior  to  re-entering  a  high-fidelity  simulation. 

Trainees  presented  with  extremely 
difficult  problems  that  appear 
unsolvable  should  be  assisted  in 
making  some  consistent  progress 
during  training. 

(Individual  Level) 

The  structure  of  the  training  environment  and  practice 
opportunities  for  team  adaptability  should  not  be  "sink  or  swim" 
(esp.  during  initial  stages  of  practice).  Feedback  and  direction 
that  actively  guides  teams  through  how  to  think  through  a 
complex  task  and  make  decisions  about  resources  is  a  critical 
foundation  of  team  adaptability  training.  Providing  guidance  that 
prompts  teams  to  explore  options  for  task  completion  during 
training  helps  to  avoid  discouragement,  anxiety,  and 
abandonment  of  effort. 

•  Use  triggers  and  backup  triggers  during  simulations  to 
allow  learners  to  attempt  the  behavior  and,  if 
unsuccessful,  observe  an  "expert"  (confederate)  execute 
the  behavior  with  success. 

•  Junior  learners  that  may  lack  clinical  knowledge  should 
be  encouraged  to  seek  assistance  for  help  at  any  time. 
Using  confederates  as  "mentors"  can  not  only  assist 
learners  through  difficult  tasks  but  also  will  build  comfort 
with  seeking  help  from  other  team  members  and  those 
outside  the  team. 

Variability  in  practice  trials  should 
be  provided  during  training  to 
maximize  retention  &  transfer. 
(Individual  and  Team  Level) 

Whereas  early  stages  of  training  are  enhanced  by  repetition  and 
rehearsal  (i.e.,  developing  declarative  &  procedural  knowledge), 
advanced  stages  of  training  are  enhanced  by  exposing  trainees 
to  as  diverse  an  array  of  scenarios  in  which  to  apply  their  KSAs 
as  possible.  It  is  particularly  critical  to  expose  trainees  to 
situations  where  previously  learned,  frequently  used,  and/or 
typically  reliable  courses  of  action  are  ineffective.  Providing 
variability  in  practice  trials  promotes  the  development  of  broader 
associative  knowledge  structures  and  contingency-based 
thinking. 

•  Use  EBAT  to  build  simulations  that  contain  appropriate 
task  complexity 

•  Shorten  intervals  between  prompts  to  increase  time 
pressures  as  appropriate. 

•  Use  confederates  to  add  interpersonal  challenges. 

•  Build  in  environmental  challenges  (e.g.,  additional 
patients,  equipment  failure)  to  increase  complexity 

Training  should  be  permissive  of, 
embrace,  and  even  encourage 
errors  made  by  learners  during 
training. 

(Individual  and  Team  Level) 

Errors  are  an  inevitable  component  of  real-world  performance. 
Errorless  training  leads  to  effective  training  performance,  but  is 
often  related  to  poor  training  transfer.  Although  errors  during 
training  should  be  brought  to  learners'  attention,  learning  that  is 
focused  on  error  management  as  opposed  to  error  prevention  is 
more  successful.  Framing  training  as  an  opportunity  to  make 
and  learn  from  errors  encourages  trainees  to  develop  problem¬ 
solving  or  hypothesis-testing  skills  and  strategies  for  managing 
affective  responses  (e.g.,  frustration  and  anxiety). 

•  Use  confederates  to  “force”  errors  during  simulations. 

This  requires  considerable  expertise  in  debriefing  to 
ensure  learners  do  not  feel  “tricked”.  Appropriate  pre¬ 
briefing  and  establishment  of  a  learning  environment  can 
help.  Be  sure  that  “errors”  meet  a  minimum  level  of 
psychological  fidelity  for  learners. 
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Incorporate  lessons  on  how  to  alter 
coordination  strategies  in  training. 
(Team  Level) 

When  task  demands  are  low,  trainees  should  learn  to  discuss 
possible  problems  that  could  arise  later  in  the  task.  By 
discussing  their  coordination  strategies  during  this  period,  they 
will  likely  reduce  the  amount  of  communication  necessary  to 
achieve  successful  team  performance  later  and  allow  them  to  be 
adaptive  when  novel  problems  arise  in  the  environment. 

•  Encourage  learners  to  develop  contingency  plans 

•  Discuss  team  member  understanding  and  mental  model 
development  during  debriefing  to  help  reinforce  the 
importance  of  discussing  and  practicing  team 
coordination 

Integrate  metacognitive  prompts 
into  training. 

(Individual  Level) 

Metacognition  is  the  process  of  actively  reflecting  on  one's 
thought  processes.  Encouraging  metacognitive  activity  during 
training  can  help  learners  identify  and  focus  on  the  goals, 
assumptions,  and  strategies  guiding  their  decision-making  and 
task  performance.  This  is  especially  important  for  less 
experienced  trainees  learning  to  perform  in  complex  and 
dynamic  environments  and  who  may  struggle  with  such  “big 
picture”  thinking. 

•  Employ  “think  aloud”  protocols  during  simulation-based 
training  in  which  the  trainee  verbalizes  their  thought 
process  during  practice 

•  Build  in  opportunities  for  more  frequent  huddles  during 
simulation-based  training  in  which  the  trainee  is 
prompted  to  explicitly  discuss  their  rationale  for  previous 
decisions  and  considerations  for  future  plans. 
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Appendix  2.  Identifying  Task  Complexity  and  Associated  Best  Practice  Training  Principles 


Adapting  to  changes  in  Component 
complexity 

Changes  in  number  and/or  difficulty  of  tasks 

Adapting  to  changes  in  Coordinative  complexity 

Changes  in  sequencing,  prioritization,  & 
interdependence  among  tasks 

Adapting  to  changes  in  Dynamic  complexity 

Volatility  in  component  &  coordinative  complexity 
within  a  task 

Principle 

Rationale 

Principle 

Rationale 

Principle 

Rationale 

Trainees  should  not 
be  provided 
strategy  instruction 
until  later  in  training 

Emphasizing  "chunking" 
and  how  to  complete  small 
numbers  of  simple, 
manageable  tasks  during 
early  knowledge/skill 
acquisition  promotes  self- 
efficacy  and  draws  focus 
away  from  premature 
comparative  &  normative 
evaluations 

Trainees  should  not 
be  provided 
strategy  instruction 
until  later  in  training 

Shifting  training  towards 
prioritization,  how  to  develop 
contingencies,  and  managing 
distal  vs.  proximal  goals  once 
trainees  have  achieved 
proficiency  in  basic  knowledge 
and  skill  promotes  mastery 
learning  and  promotes  "big 
picture"  thinking 

Trainees  should  not 
be  provided 
strategy  instruction 
until  later  in  training 

Shifting  training  towards 
recognizing  when  change  is 
needed  and  when/how  to 
implement  contingencies 
focuses  trainees  appropriately 
on  normative  expectations  and 
being  proactive. 

Training  material 
should  be 
structured  so  that 
instruction  proceeds 
from  general  to 
detailed,  specific  to 
complex 

Scaffolding  practice  such 
that  trainees  learn  to  deal 
with  few/simple  tasks  --> 
more/simple  tasks  — > 
few/difficult  tasks  --> 
more/difficult  tasks  enables 
training/feedback  to  focus 
on  quantity  vs.  complexity 
of  tasks,  which  pose 
different  considerations 

Training  material 
should  be 
structured  so  that 
instruction  proceeds 
from  general  to 
detailed,  specific  to 
complex 

Scaffolding  practice  such  that 
trainees  learn  to  deal  with 
few/simple  tasks  — > 
more/simple  tasks  --> 
few/difficult  tasks  — > 
more/difficult  tasks  enables 
training/feedback  to  focus  on 
quantity  vs.  complexity  of 
tasks,  which  pose  different 
considerations 

Training  material 
should  be 
structured  so  that 
instruction  proceeds 
from  general  to 
detailed,  specific  to 
complex 

Training  that  allows  practice 
shifting  from  few/simple  tasks 
to  more/complex  tasks  within 
the  learning  environment 
allows  learners  to  practice 
situation  assessment  and  task 
regulation  cycles  under 
different  demands 

Trainees  learning  a 
complex  task 
should  be 
encouraged  to 
monitor  rate  of 
learning  progress 
rather  than  just 
learning 
performance 

Focusing  feedback  on  how 
and  what  KSAs  trainees 
have  developed  that 
involve  managing 
different  quantities  of 
tasks  minimizes  goal 
abandonment  and 
promotes  learning  how  to 
deal  with  situations  where 
resources  (time,  persons, 
etc.)  are  strained 

Trainees  learning  a 
complex  task 
should  be 
encouraged  to 
monitor  rate  of 
learning  progress 
rather  than  just 
learning 
performance 

Focusing  feedback  on  how  and 
what  KSAs  trainees  have 
developed  that  involve 
managing  tasks  with  fewer 
vs.  more  interdependencies 
and  considerations  minimizes 
goal  abandonment  and 
promotes  learning  how  to  deal 
with  situations  where 
resources  must  be  highly 
coordinated 

Trainees  learning  a 
complex  task 
should  be 
encouraged  to 
monitor  rate  of 
learning  progress 
rather  than  just 
learning 
performance 

Focusing  feedback  on  how 
and  what  KSAs  trainees  have 
developed  that  are  involve 
managing  sudden  changes 
in  task  demands  minimizes 
goal  abandonment  and 
promotes  learning  how  to  deal 
with  situations  where 
resources  must  be  quickly 
assessed,  gathered,  and 
distributed 

Provide  & 

emphasize  proximal 
subgoals  that  allows 
trainees  to  break 
task  down  into 
manageable 
components 

Focusing  on  how  to  deal 
with  multiple  competing 
demands  and  strained 
resources  improves 
capacity  to  manage  tasks 
where  demands  >=  supply 

Provide  & 

emphasize  proximal 
subgoals  that  allows 
trainees  to  break 
task  down  into 
manageable 
components 

Focusing  on  how  to  prioritize 
and  structure  task  activity 
improves  capacity  to  make 
informed  decisions  & 
communicate  what  must  be 
accomplished  to  reach  task 
goals 

Provide  & 

emphasize  proximal 
subgoals  that  allows 
trainees  to  break 
task  down  into 
manageable 
components 

Focusing  on  how  to  deal  with 
variability  in  task 
demands/resources  within  a 
single  performance  event 
improves  capacity  to  shape 
and  implement  contingencies 
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Adapting  to  changes  in  Component 
complexity 

Changes  in  number  and/or  difficulty  of  tasks 

Adapting  to  changes  in  Coordinative  complexity 

Changes  in  sequencing,  prioritization,  & 
interdependence  among  tasks 

Adapting  to  changes  in  Dynamic  complexity 

Volatility  in  component  &  coordinative  complexity 
within  a  task 

Principle 

Rationale 

Principle 

Rationale 

Principle 

Rationale 

Variability  in 
practice  trials 
should  be  provided 
during  training  to 
maximize  retention 
&  transfer 

Practicing  multiple 
situations  with 
fewer/simple, 

fewer/difficult,  more/simple, 
more/difficult  exposes 
trainees  to  more 
exemplars,  prepares  them 
for  more  situations,  and 
encourages  flexible  modes 
of  thinking/problem-solving 

Variability  in 
practice  trials 
should  be  provided 
during  training  to 
maximize  retention 
&  transfer 

Practicing  multiple  situations 
with  fewer/simple, 
fewer/difficult,  more/simple, 
more/difficult  exposes  trainees 
to  more  exemplars,  prepares 
them  for  more  situations,  and 
encourages  flexible  modes  of 
thinking/problem-solving 

Variability  in 
practice  trials 
should  be  provided 
during  training  to 
maximize  retention 
&  transfer 

Practicing  situations  that 
transition  from  fewer/simple, 
fewer/difficult,  more/simple, 
more/difficult  within  the 
learning  environment  exposes 
trainees  to  more  exemplars, 
prepares  them  for  more 
situations,  and  encourages 
flexible  modes  of 
thinking/problem-solving 

Trainees  should  be 
encouraged  to 
experience  errors 

Errors  of  omission  & 
commission  are  common 
stimulus  for  adaptation. 
Placing  trainees  in 
situations  where  few  vs. 
many,  little  vs.  big,  salient 
vs.  subtle,  etc.  errors  are 
likely  and/or  have 
happened  reinforces 
situation  awareness  and 
decision-making  skills  in 
unexpected  and  unplanned 
situations 

Trainees  should  be 
encouraged  to 
experience  errors 

Errors  of  omission  commission 
are  common  stimuli  for 
adaptation.  Placing  trainees  in 
situations  where  errors  push 
them  down  a  wrong  path 
reinforces  situation  awareness 
and  decision-making  skills  in 
unexpected  and  unplanned 
situations 

Trainees  should  be 
encouraged  to 
experience  errors 

Errors  of  omission  & 
commission  are  common 
stimuli  for  adaptation.  Placing 
trainees  in  situations  where 
tasks  change  suddenly  and 
errors  are  more  likely 
reinforces  situation  awareness 
and  decision-making  skills  in 
unexpected  and  unplanned 
situations 
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Appendix  3.  Principles  of  providing  adaptive  feedback 

Principle  1.  Trainees  should  be  provided  with  accurate  and  credible  feedback. 

Ensuring  feedback  is  accurate  helps  trainees  understand  what  task  behaviors  need  improvement.  Making 
feedback  credible/authentic  improves  the  likelihood  that  trainees  perceive  the  feedback  as  something 
important  to  which  they  should  attend.  There  are  instances  in  which  the  accuracy  of  feedback  should  be 
"altered"  if  it  benefits  self-efficacy  and  effort  of  trainees  (e.g.,  learning  a  complex  task  that  results  in  many 
mistakes,  poor  training  performance,  etc.). 

Simulation  Recommendations: 

•  Explain  learning  objectives  to  trainees  and  explain  clear  benchmarks  for  performance.  By  setting 
benchmarks,  trainees  can  see  where  their  performance  gaps  lie.  Setting  benchmarks  also  helps  ensure 
feedback  is  diagnostic. 

•  The  feedback  facilitator  should  have  significant  skill  in  debriefing  techniques. 

•  Consider  pairing  a  content  expert  with  feedback  expert  when  needed 

Principle  2.  The  frequency  and  timing  of  feedback  should  be  appropriately  tailored  to  trainees  and  the 
goal  of  training. 

In  general,  directive,  immediate,  and  frequent  feedback  tends  to  facilitate  the  acquisition  of  declarative  & 
procedural  knowledge  and  improve  learner's  self-efficacy.  However,  when  the  goal  of  training  is  to  promote 
how  to  identify  and  handle  errors  and/or  develop  strategies  and  contingency-based  thinking,  feedback  should 
be  less  frequent  to  discourage  trainees  from  assuming  there  is  "one  correct  answer"  they  should  be  learning. 

Simulation  Recommendations: 

•  Process  feedback  should  be  more  frequent  than  outcome  feedback 

•  With  more  experienced  teams,  moving  from  a  formalized  feedback  to  facilitation  of  a  high-level  debrief  that 
allows  objectives  to  emerge  based  on  performance  and  team  challenges  might  be  more  appropriate 

•  When  performing  a  more  high-level  debrief,  it  should  occur  as  close  to  the  event  as  possible 

•  Be  sure  to  build  in  adequate  time  for  debriefs,  usually  a  minimum  of  2x  the  length  of  the  simulation 

•  Ensure  that  the  simulation  objectives  are  finite  and  can  be  covered  during  the  debrief 

Principle  3.  Feedback  related  to  practice  behaviors  and  strategy  development  should  be  specific. 

When  it  is  appropriate  to  provide  such  feedback  (see  principle  above),  feedback  about  the  behaviors  in  which 
trainees  engaged;  how,  why,  and  what  strategies  trainees  attempted  to  implement;  and  the  manner  by  which 
they  addressed  errors  or  unexpected  events  should  be  specific  and  detailed.  Providing  specific  feedback 
facilitates  the  retention  and  automatizing  of  learned  material  and  helps  to  avoid  ineffective  strategy  or 
behavioral  changes. 

Simulation  Recommendations: 

•  Ensure  that  team  members  have  a  working  knowledge  of  team  processes  prior  to  executing  the  simulation; 
this  will  allow  the  facilitator  to  use  this  common  language  during  the  debrief 

•  Refer  to  specific  examples  during  the  simulation  to  highlight  strengths  and  weaknesses  of  team  process. 

•  Video  review  may  be  helpful 

•  Providing  individuals  with  feedback  is  important;  however,  must  be  done  with  care  in  a  team  debrief 

•  Using  self-assessment  "cognitive  aids"  can  help  individuals  assess  their  contribution  to  team  performance. 

Principle  4.  Feedback  should  be  more  heavily  focused  towards  process  rather  than  outcome. 

Outcome  feedback  conveys  the  extent  to  which  trainees  met/are  meeting  learning  objectives.  Alternatively, 
process  feedback  focuses  on  how  trainees  are  using  information,  performing  behaviors,  and  the  steps  used  to 
complete  task  activities.  Process  feedback  directs  learners  to  reflect  on  the  strategies  and  decisions  that  led  to 
particular  outcomes,  and  is  thus  particularly  important  when  the  goal  of  training  is  to  improve 
regulatory/strategic  thinking. 

Simulation  Recommendations: 

•  Allow  teams  to  discuss  medical  content  and  address  any  concerns  quickly  to  help  learners  focus  on 
processes  of  care 

•  Encourage  learners  to  consider  other  circumstances  where  similar  processes  are  employed  and  can  fail. 

This  helps  team  focus  on  processes  instead  of  the  specific  clinical  issues  presented  in  the  simulation. 
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Principle  5.  Trainees  should  be  encouraged  to  believe  substantial  negative  performance  discrepancies 
are  moderate. 

Acquiring  KSAs  in  complex  task  environments  is  challenging,  and  learners  are  not  likely  to  perform  well  during 
initial  stages  of  training.  Providing  accurate  and  credible  feedback  is  important,  but  it  is  equally  critical  to 
ensure  that  trainees  do  not  become  overwhelmed  and/or  discouraged  by  actions  they  have  performed 
incorrectly.  This  balance  can  be  achieved  by  framing  feedback  such  that:  (1)  feedback  emphasizes  trainee 
performance  is  attributable  to  controllable  factors;  (2)  feedback  de-emphasizes  outcome-focused  feedback  in 
favor  of  process  feedback  and  feedback  that  highlights  how  learners  are  developing;  (3)  initially  poor 
performance  be  labeled  as  only  moderately  negative.  Doing  so  decreases  the  likelihood  of  goal  abandonment 
while  increasing  the  likelihood  that  effort  and  self-efficacy  will  be  maintained. 

Simulation  Recommendations: 

•  Encourage  learners  to  note  positive  as  well  as  negative  behaviors  (What  should  you  change?  What  should 
you  do  the  same?) 

•  Encourage  learners  to  see  how  even  effective  processes  can  result  in  poor  outcomes 

•  Limit  the  focus  of  the  debrief  to  just  learning  objectives  to  avoid  talking  about  too  many  issues 

•  Focus  on  process,  not  outcomes 

Principle  6.  The  provision  of  negative  and/or  normative  feedback  should  be  minimized  to  trainees 
learning  a  complex  task. 

Negative  feedback  (i.e.,  learners  are  failing  to  meet  learning  objectives)  and  normative  feedback  (i.e. , 
comparing  learners  to  an  external  standard)  tends  to  shift  trainees'  attributions  towards  the  self  &  ego 
protection,  which  generally  interferes  with  the  acquisition  of  KSAs.  Negative  feedback-especially  when 
learning  a  complex  task--is  demotivating  and  tends  to  decrease  self-efficacy.  In  general,  positive  performance 
feedback  tends  to  improve  self-efficacy,  though  it  must  be  accurate  and  credible  to  prevent  complacency 
and/or  disengagement. 

Simulation  Recommendations: 

•  Provide  a  supportive  climate  that  allows  participants  to  share  opinions  openly  and  honestly 

•  Critical  step,  as  learners  cite  a  fear  of  educator  and  peer  judgment  as  barrier 

•  Use  "good  judgment"  framework  or  advocacy/inquiry  to  discuss  negative  performance  and  uncover  learner 
mental  models  and  frames  that  are  supporting  suboptimal  performance 

Principle  7.  Guidance  that  directs  trainees  to  consider  what  they  should  think  about  and  how  to  think 
about  it  should  be  provided  to  trainees  in  learner  control  environments. 

Guidance  is  a  proactive  "feed-forward"  mechanism  that  encourages  learners  to  take  an  active  role  in 
considering  how  and  why  they  are  engaging  in  particular  learning  behaviors.  Guidance  promotes  learning 
through  both  increased  metacognition  (i.e.,  "thinking  about  thinking")  and  encouraging  an  exploratory/future- 
focused  perspective  on  learning-both  of  which  are  critical  conditions  for  learning  complex  tasks  and  strategies. 
There  are  many  options  for  what  type  of  guidance  can  be  provided,  but  typical  categories  include  focusing 
trainees  on  how  and  where  to  direct  attention  during  training  (cognition),  manage  effort  and  emotions  (affect), 
and  sequence  actions  (behaviors). 

Simulation  Recommendation: 

•  Learners  should  be  encouraged  to  identify  their  strengths  and  weaknesses.  With  instructor  input,  this 
information  should  be  used  to  guide  training  content  and  emphasis.  In  this  way,  learners  can  focus  on  more 
basic  skills  where  they  need  development  and  challenge  themselves  in  areas  where  they  excel. 

•  Guidance  can  also  come  in  the  form  of  affect/error  regulation  that  emphasizes  to  learners  that  good 
processes  don't  always  result  in  good  outcomes. 

Principle  8.  Match  the  level  of  feedback  provided  to  the  level  of  the  goals  in  training. 

Feedback  provided  in  training  directs  individuals  to  allocate  resources  and  perform  self-regulation  activities  in 
relation  to  specific  goals.  However,  trainees  can  have  goals  across  multiple  levels  thereby  complicating 
trainees'  decisions  about  which  goals  to  strive  toward.  Therefore,  if  the  focus  of  training  is  to  achieve 
individual-level  goals,  feedback  providers  should  provide  individual-level  feedback  so  resources  are  directed  to 
individual  goal  attainment.  Similarly,  if  trainees  should  focus  on  team-level  goals,  feedback  providers  should 
provide  team-level  feedback  to  direct  resources  toward  team  goal  attainment. 
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Simulation  Recommendations: 

•The  debriefing  plan  should  be  pre-planned  and  should  target  appropriate  level(s)  based  upon  learning 
objectives. 

•  When  individual  feedback  is  necessary  within  a  team  context,  the  learner  should  be  approached  separately  if 
there  is  an  issue  with  individual  clinical  competence  or  procedural  skills. 

•  If  individual  feedback  on  a  team  skill  is  necessary,  feedback  should  be  framed  as  a  team-based  learning 
point. 
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Appendix  4.  Examples  of  candidate  variables  for  the  BBN  model 


Examples  of  candidate  team  process  variables  for  BBN  model 


Process 

Item 

Activity 

MA 

Nurse  is  questioned  to  gather  additional  information  (e.g.,  “Did  the  EMS  leave  a  run  sheet?”) 

Evaluation 

MA 

Reason  for  patient  admittance  is  communicated 

Evaluation 

MA 

Pre-hospital  vital  signs  are  communicated 

Evaluation 

GS 

Request  for  updated/new  vital  signs  is  given  (versus  pre-hospital  vitals) 

Evaluation 

MA 

Updated  vital  signs  are  communicated  to  team 

Evaluation 

GS 

Pulse  checked  on  arm  or  neck  of  patient 

Evaluation 

GS 

Pulse  checked  in  feet 

Evaluation 

GS 

Breathing  checked  by  listening  to  chest  with  stethoscope 

Evaluation 

MA 

Results  of  lung  exam  are  communicated  (e.g.,  “breathing  is  labored”) 

Evaluation 

GS 

Gag  reflex  checked 

Evaluation 

GS 

Attempts  to  elicit  speech  by  talking  to  patient 

Evaluation 

GS 

Chest  wall  is  assessed  by  palpation 

Evaluation 

MA 

Glasgow  coma  scale  (GCS)  is  communicated  (e.g.,  “GCS  is  XXX”) 

Evaluation 

MA 

Abdomen  is  assessed 

Evaluation 

TM/BB 

Blood  pressure  is  monitored  and  communicated  during  intubation 

Airway  management 

SM 

ET  tube  placement  is  verified  by  listening  to  patient’s  chest 

Airway  management 

MA 

Team  discusses  cause(s)  of  hypotension 

Injury  management 

SM 

Team  members  monitor  blood  pressure  during  blood  transfusion 

Injury  management 

Examples  of  candidate  team  performance  variables  for  BBN  model 


Performance 

Category 

Item 

Diagnosis 

Cardiac  monitor  interpretation 

HR  is  verbalized 

Rhythm  is  assessed  to  be  “tachycardia” 

GCS 

Correctly  calculated 

Evaluation 

Radiographs  Ordered 

Time  CXR  ordered 

Time  Head  CT  ordered 

Time  Pelvis  X  Ray  ordered 

Time  FAST  or  USN  ordered 

Airway  management 

Intubation 

Length  of  intubation  (time) 

Time  decision  made  to  intubate 

Bagging  rate  immediately  following  intubation 
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Development  of  an  Integrated  Team  Training  Design  and  Assessment 
Architecture  to  Support  Adaptability  in  Healthcare  Teams 

MSIS-Team  Performance  Training  Research  Initiative 

PI:  R.  Fernandez  /  J.  Grand  Org:  University  of  Washington 


Problem,  Rationale,  and  Military  Relevance 

Problem:  Conceptual  models  and  assessment  approaches  to 
support  effective  team  training  that  maximizes  team  adaptability 
and  performance  do  not  exist. 

Rationale:  An  integrated  team  training  model  will  identify  which 
individual,  team,  and  training  design  factors  can  be  manipulated 
to  maximize  team  training  effectiveness  and  impact  on  patient 
safety  outcomes.  Additionally,  a  predictive  model  of  team 
performance  will  demonstrate  bow  team  behaviors  predict  future 
team  performance  and  patient  care  outcomes. 

Military  Relevance:  This  proposal  directly  addresses  the  TPT 
research  initiative  by  providing  a  detailed  framework  and 
predictive  assessment  system  to  support  team  performance 
training  to  improve  teamwork  behaviors  and  patient  outcomes. 


Conceptual  Model  of 
^Team  Training  Elements 


+ 

T 


Predictive  Model  of 
Team  Performance 


OPTIMIZED  TRAINING 


IMPROVED  ADAPTABILITY  AND 
PATIENT  SAFETY 


Proposed  Solution 

Objective:  To  develop  a  simulation  design  architecture  and 
predictive  model  of  trauma  team  performance  to  support  team 
training  and  team  effectiveness. 

Summary  of  Aims:  Integrate  individual-  and  team-level  team 
performance  frameworks  to  develop  a  simulation  design 
architecture  and  a  predictive  model  of  trauma  team 
performance  to  support  effective  team  training  with  automated 
individual  and  team  feedback  and  performance  assessment. 

Outcomes:  (1 )  A  detailed  framework  of  the  individual,  team, 
and  training  design  factors  related  to  effective  team 
performance  training  and  (2)  A  predictive  model  of  team 
performance  that  identifies  how  teams  can  adapt  their 
behaviors  to  maximize  their  teamwork  and  minimize  errors 


Timeline  and  Cost 


Activities  FY 

15 

16 

Integrate  individual-level  and  team-level  simulation 
design  frameworks  to  develop  a  simulation  design 
architecture  (Aim  1) 

■ 

Develop  a  predictive  model  of  trauma  team 
performance  and  outcomes  using  Bayesian  Belief 
Networks  (Aim  2) 

— 

Prospectively  test  and  refine  the  model  of  trauma 
team  performance  on  simulated  trauma  team 
resuscitations  (Aim  2) 

Data  analysis  and  dissemination 

Estimated  Budget  ($K) 

591 K 

556K 

